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The processes used when developing a survey instrument for research studies are equally as important 
when developing a survey for evaluation purposes. This paper presents the steps involved in the 
development and validation of a survey instrument that can be used to evaluate online training modules or 
instructional aids in both workplace and educational environments. The resulting survey measures the 
effectiveness, appeal, and efficiency of online training materials. 
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The use of computer-based and online training has soared as advances in technology have made it easier for 
instructors, trainers, and curriculum developers to put course materials online. As in traditional classroom 
environments, evaluation of materials is important, and the online environment adds another dimension to 
evaluation. However, most articles on online training materials fall short when it comes to student or participant 
evaluation of those materials. Few studies are grounded in the use of valid and reliable instruments to evaluate 
online training materials. Michalski and Cousins (2001) note that training providers often view evaluation as having 
mixed purposes, both instrumental and symbolic, that are used “mostly to highlight training merit and worth and to 
sustain and expand training budgets” (p. 37). 

This study attempts to address this issue by developing an instrument to measure constructs associated with the 
use of online training materials, or instructional aids. This instrument was developed and pilot-tested in an online 
health occupations science course in kinesiology. In this course, students used instructional aids such as videos on 
movements of the body and muscle testing, digital flash cards and games to identify key concepts related to muscles 
of the body, and a tutorial program to review content on osteology. These instructional aids were placed within the 
Blackboard Learning Management System for review and practice of content in preparation for exams. Giguere, 
Formica, and Harding (2004) note the great number of studies exploring online learning in academia, and also the 
lack of studies exploring online methodologies for professional development training. They believe that “training 
and technical assistance providers, who are often responsible for providing professional development and continuing 
education activities . . . can benefit from the lessons learned from the use of these strategies in higher education 
settings” (p. 208). While the instrument was developed for use in formal education settings, it is applicable for use in 
any organization employing online learning materials, including business and non-profit sectors. 

Instructional aids are defined as small units of digital educational materials that can be used flexibly and in a 
variety of formats (e.g., videos, interactive games, and tutorials) to enhance online lessons. Also known as learning 
objects, instructional aids “decompose content into granular pieces of information that can be stored, retrieved, and 
reused in instruction” (Jonassen & Churchill, 2004, p. 32). These learning objects or instructional aids can be used 
individually, or they can be linked together in units to form a course (Hamel & Ryan-Jones, 2001). 

Instructional aids are being used increasingly in human resource development and online education as teaching 
tools to help students understand concepts. They are easy to use, and can be reused in different contexts (making 
them cost efficient) (Concei9ao, Olgren, & Ploetz, 2006). Because of this flexibility, it has been suggested that they 
are the future of online instruction. Hamel and Ryan-Jones (2001) posit that instructional designers “will not be 
designing courses anymore. [Rather] they will be designing small stand-alone units of instruction called learning 
objects” (p. 1058). The successful use of instructional aids for learning incorporates the following constructs: 
effectiveness, efficiency, and appeal (Reigeluth, 1999). Effectiveness is how well the instructional aids work relative 
to student learning. Efficiency is defined by the level of effectiveness of the instruction divided by the time of the 
instruction. The level of appeal is the extent to which the learners enjoy using the instructional aids. 
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The constructs of effectiveness, efficiency, and appeal take on new dimensions in an online environment. 
Pollack (1998) notes that “evaluation is frequently difficult in face-to-face learning environments and is 
compounded in distance education” (p. 6). This study adds to the body of knowledge on evaluation by developing 
and validating an evaluation instrument including the constructs defined by Reigeluth (1999), while taking into 
account the online environment. Using a valid and reliable instrument that measures all three constructs in an online 
environment will help instructional designers and trainers to know more accurately if the online instructional aids 
their students are using are, in fact, enhancing learning. 

Theoretical Framework 

The theoretical framework used in the development and validation of this survey is based on the work of Wallen and 
Fraenkel (2001), Rae and Parker (1997), Dillman, Tortora, and Bowker (1998), Dillman and Bowker (2001), and 
their contributions to survey development. Wallen and Fraenkel’s process includes the definition of the problem, 
identification of the target population, determination of mode of data collection, preparation of the instrument, 
collection of data, and analysis. Also important to their process is the identification of large categories of issues (or 
constructs), which can be used to suggest more specific issues within each category, which can then be used for 
question development. Rae and Parker’s (1997) stages of the survey research process, which emphasize the 
importance of piloting or pretesting the survey instrument, then revising the survey based on pretest results, were 
also used in theoretical framework development. Works by Dillman et al. (1998) and Dillman and Bowker (2001) 
form the theoretical bases for the online components of this work. These authors have researched the topic of survey 
development specifically in online environments. Research by Strachota, Schmidt, and Conceiqao (2006) on online 
survey development, specifically addressing steps in survey development and environmental factors associated with 
online survey development, was also used as a theoretical base. The basis for the evaluation component of this 
study, including the validation of constructs, is based on the theoretical work by Reigeluth (1999). 

Research Questions and Propositions 

This study addresses the following research questions: 

• What are the steps associated with the development of an instrument to evaluate online instructional aids? 

• How can training instructional aids best be evaluated by users? 

• What constructs best evaluate instructional aids in online environments? 

The researchers propose that the same steps that are used in the development of a survey for research purposes 
can be used in the development of a survey for evaluation of online training modules or instructional aids. 

Research Design 

A model for online survey development was used to develop and validate the evaluation tool described in this paper 
(see Figure 1) (Schmidt, Strachota, & Conceiqao, 2006). This model starts with the review of literature, the 
development of a research question, survey construct development, and then survey question development. The 
validation processes described in this model require a pilot test, factor analysis, and question revision as appropriate. 
The survey is then evaluated for online use. This model was used because it stresses the importance of examining 
both general environmental concerns associated with survey research, as well as factors specifically related to the 
use of surveys in online environments. 

Methodology 

A review of literature on instructional aids was conducted, and it was determined that there was a gap in the 
literature on reliable and valid instruments used to evaluate instructional aids in business and higher education 
settings. Constructs related to the evaluation of instructional aids were identified based on Reigeluth’s (1999) 
instructional design theory. It was determined that the constructs of effectiveness, efficiency, and appeal could best 
be used in evaluation in an online environment. Based upon these constructs, survey questions were developed by 
the researchers. To ensure content validity, experts in survey research and in the subject matter were consulted and 
their input was used in question revision and redesign. A pilot test (N=20) was then conducted in order to establish 
content validity. 
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Figure 1. Model for Online Survey Development 



The pilot test was conducted in a kinesiology online course at a technical college. This study was conducted 
during the spring 2006 semester. The structure of the course included 1 5 modules of online instruction, optional one- 
hour open labs held each week when there was a scheduled on-campus proctored exam, online discussion groups, 
and online quizzes. The open lab was designed primarily as a review of content and for hands-on demonstration. 
Students were given surveys asking their opinion of the effectiveness, efficiency, and appeal of the instructional aid 
created for each module. The survey instrument was pilot tested with the instructional aid developed for module one. 
Modifications were made to the survey instrument based on the pilot study. The revised instrument was then used 
for evaluation of the remaining instructional aids. 

Twenty students enrolled in the online course completed a survey instrument to evaluate each instructional aid 
embedded within each lesson during the course of the semester. Students were given these surveys after each unit of 
instruction. The survey was customized for each instructional aid used during the unit. For example, for Unit 1 the 
instructional aid was video, so a survey with questions specific to the video aid was conducted. Units 2 and 3 used a 
tutorial, so surveys focusing on tutorials were conducted. Units that contained interactive games used surveys 
specific to each of these instructional aids. All survey questions were the same, with the exception of the type of 
instructional aid noted on the survey. 

Survey Instrument 

As a result of the pilot test, the instructional aids for modules two through fifteen of the Kinesiology course 
were evaluated using the following final survey instrument. 

Instructional Aids Survey Instrument 





The purpose of this investigation is to study the effectiveness, efficiency, and appeal of instructional aids within this 
college course. Data collected from this survey will be kept confidential. Data will be grouped and your comments 
will not be individually identifiable. Filling out this survey indicates that you are at least 18 years old and are giving 
your informed consent to be a participant in this study. 

DEMOGRAPHICS 

Age: _18-24 _25-34 _35-44 _45-54 _55-64 
Gender: Male Female 

Ethnicity: Caucasian/White, African American, Hispanic, Asian, Native American 



Strongly Agree 


Agree 


Neutral 


Disagree 


Strongly Disagree 



EFFECTIVENESS OF INSTRUCTIONAL AIDS 

1. This instructional aid helped me when practicing (insert as appropriate: movements of the body, muscle testing 
positions, etc). 

2. This instructional aid helped me to better understand the information (insert as appropriate: movements of the 
body, muscle testing positions, etc). 

3. This instructional aid put meaning to the written material (content) for this lesson. 

4. This instructional aid helped me to better understand the textbook information. 

APPEAL OF INSTRUCTIONAL AIDS 

5. This instructional aid was organized by a specific (insert as appropriate: joint, bone, muscle) so that it was easy to 
search and review information. 

6. This instructional aid offered feedback so I knew if my response was correct or incorrect and if I needed to 
continue to review. 

7. 1 was satisfied with the design of this instructional aid. 

8. 1 was satisfied with the look of this instructional aid (visual clarity). 

EFFICIENCY OF INSTRUCTIONAL AIDS 

9. Approximately how many hours did you spend reviewing the information in this instructional aid? 

Hours 

10. Select the purpose for using this instructional aid (select all that apply): 

Reinforce, Clarification, Practice, Review, Retention 

1 1 . 1 experienced technical difficulty when using this instructional aid? Y es No 

If yes, describe what technical problems you encountered. 



GENERAL SATISFACTION WITH INSTRUCTIONAL AIDS 



Strongly Agree 


Agree 


Neutral 


Disagree 


Strongly Disagree 



12. Did you like using the instructional aids in this course? 

13. Overall I was satisfied with this instructional aid. 

14. Overall I feel I was able to learn the information from this instructional aid as well as I would have in a face-to- 
face class presentation. 

Reliability and Validity 

Validity and reliability were established for the instrument as part of the pilot testing process. Through the use 
of factor analysis using varimax rotation the instrument identified two strong constructs: effectiveness and appeal. 
Items within the construct of effectiveness factor loaded from .800 to .916. Items within the construct of appeal 
factor loaded from .618 to .905. Reliability was found to have a Cronbach’s alpha of .90. Efficiency was found not 
to be a strong construct because of its interdependency on effectiveness. It was determined that efficiency could be 
more accurately measured through the use of qualitative questions. Therefore, qualitative questions were designed 
for the efficiency category. The pilot test was found to be an important stage of the instrument development process 
as necessary revisions were identified. These revisions were made and the final survey instrument was used to 
evaluate the instructional aids used for modules two through fifteen. Analysis of the final survey instrument using 
the two constructs of effectiveness and appeal showed strong factor loading of all questions within the appropriate 
construct and a Cronbach’s alpha of .89 for reliability. 





Survey Results 



Survey results showed that students were highly satisfied with the instructional aids used in this online course. There 
was no significant difference in the level of effectiveness and appeal when comparing the use of instructional aids 
such as video, interactive games, and tutorial instruction. Overall video received the highest ratings compared to 
interactive games and tutorials. Video was used for the movements of the body and for all muscle testing units in the 
course. Through qualitative analysis of efficiency, students identified that video was used most often for clarification 
of concepts (81.3%). Comments included statements such as, “Viewing the video clips helps me understand the 
content better by viewing it instead of just reading the material” or “When I had a question I was able to go back to 
the videos to find out the answer and it made it more clear.” Therefore, the use of video enhanced learning by 
offering visual feedback for clarification of written content. 

Interactive games were designed using a QUIA software program, which allowed the designer to develop a 
variety of activities such as flashcards with embedded graphics and audio, matching games, concentration, challenge 
boards, ordered lists, higher-ordered questioning, and self-assessment quizzes. Through qualitative analysis of 
efficiency, students identified that interactive games were used most often to reinforce concepts (76.5%) and for 
practice (58.8%) and review (58.8%). Interactive games were used least often for clarification (29.4%). Comments 
included statements such as, “The QUIA flashcards reinforced learning for example, being able to save or discard 
cards to find out which ones I needed to practice more,” or “The flashcards help a lot. As soon as I saw the pictures 
on the cards I remembered them from the reading materials. It made studying more efficient.” The interactive 
activities were developed to reinforce written content and for practice and review as often as needed prior to the 
exam. 

A tutorial was developed for the osteology unit using Authorware software. The tutorial was created with a 
graphic of each bone and the specific landmarks of that bone. Landmark names were created as labels on the left of 
the screen. The student was to mentally attempt to identify where the landmark was located and then click on a 
designated label. Once the label was selected it then moved to the correct location on the bone. Through qualitative 
analysis of efficiency, students identified that the tutorial was used most often to reinforce concepts (75%) and for 
practice (62.5%). Comments included statements such as, “The program helped me see the landmarks of the bones. 
It helps because the bone is there and I can say what it is, then click on it and I can see if I got it right,” or “The 
tutorial was a great tool in reviewing and reinforcing all the landmarks of the bones.” The tutorial was developed to 
reinforce written content and to offer feedback to the student as to what content had or had not been learned. 

Overall the majority of students felt that they could learn online as effectively as they could in a classroom 
environment. Analysis of instructional media showed that 87.5% of the students (strongly agree and agree) felt they 
could learn online as effectively when video was used (6.3% neutral, 6.3% disagree). This compared to 70.5% of the 
students (strongly agree and agree) who felt they could learn as effectively online when interactive games were used 
(17.6% neutral, 1 1.8% disagree), whereas 65% of the students (strongly agree and agree) felt they could learn online 
as effectively when tutorial instruction (18.8% neutral, 18.8% disagree) was used. As one student stated, “With 
video, I was able to replay it, and replay it. In a classroom setting you basically see it once.” Another valuable 
comment made was, “I would like to say that all of the instructional aids that are provided with this course have 
been extremely helpful and beneficial to grasping the course content. I don’t think this course could be an online 
course (effectively) without the way it is set up organizationally and also without the tools that have been provided.” 

There are several limitations associated with this survey development process. Those limitations include the fact 
that the sample size was small, and the course used in this study was a required program course offered only online 
(which may or may not be viewed as a limitation). Another limitation is the fact that although this survey is general 
enough to be used in many different training situations, it was developed specifically in a higher education 
environment, rather than in a variety of environments. 

Conclusions and Recommendations 

This pilot study has resulted in ( 1 ) the development and validation of a survey instrument that has been thoroughly 
tested according to standards set forth for survey research studies, (2) a description of an evaluation strategy to 
evaluate training instructional aids, and (3) the identification of constructs to evaluation instructional aids in online 
environments. Surveys used for evaluation purposes are often compiled by trainers, instructional designers, and 
educators, with no bases in reliability and validity. This pilot study set out to address that issue in the context of an 
online environment, which adds another layer of complexity to the evaluation process. Using the theoretical bases 
noted in this paper, along with a model of online survey development, the researchers were able to develop a general 




survey for the evaluation of online training materials or instructional aids that can be used in business, non-profit, 
formal education, and many other settings. 

Formality of measurement process, accuracy of measurement, and pilot testing are important elements to help 
strengthen the online evaluation process. Conducting a pilot test can establish construct validity rather than simply 
having a series of questions that do not measure any given construct. The results of the pilot test in this study 
showed that the effectiveness, efficiency, and appeal (the constructs) of instructional aids did not depend on the type 
of instructional aid used. However, the use of video, interactive games, and tutorials showed the importance of 
moving beyond text in an online environment. While the use of instructional aids is important in facilitating student 
learning, the type of aid was not. In this pilot study, students used different instructional aids to accomplish different 
goals. Even though there was no significant difference in the level of satisfaction, students rated video as their first 
choice of instructional aids, followed by interactive games, and then tutorials. Conclusions can not be drawn from 
this study as there was "no significant difference" statistically. For this reason, there is a need for further research. 

It is recommended that instructional designers consider the goals of the student when choosing the type of 
instructional aid to develop. As noted above, students used video aids primarily for clarification of concepts; Hash 
cards primarily to reinforce concepts, practice, and review; and tutorials most often to reinforce concepts. 

It is also recommended that this survey be used in settings other than the one in which it was developed. For 
example, human resource development professionals may use this survey in evaluating online training materials. 
Further studies of this nature should also be conducted to support the findings of this research study. 

Research Contributes to New Knowledge In HRD 

Klecker (2005) states that “online assessments can measure the student’s achievement of intended learning objects 
if, and only if, great diligence is used in their construction” (p. 2). The formal process used in the development and 
validation of this survey instrument is an example of exactly that diligence. Pilot testing a survey instrument is a 
critical step of survey research. In this study, the survey instrument was revised based on the pilot study. This study 
serves to link theories of survey development to the evaluation of online learning, which is an important and ever- 
expanding topic in the field of human resource development. There is a dearth of research on reliable and valid 
instruments developed for evaluation purposes. This study contributes to the body of knowledge on evaluation and 
assessment of training materials. 
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